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Executive Summary 


More than 1 million students in nearly 16,000 schools participated in the spring 2014 PARCC 
field test. Fourteen states and the District of Columbia administered the test. 


The primary purposes of the PARCC field test were to: 

™ ~~ Examine the quality of test questions and tasks; 

™ Evaluate assessment training materials and administration procedures; 
™ Evaluate the computer-based delivery platform; and 
| 


Conduct research on a range of topics, including those that will inform the reporting of results from the first round 
of full testing. 


The ultimate goal of the field test was to confirm that PARCC is a quality assessment program and to make improvements based on the 
experience prior to the 2014—15 administration, in which an estimated 5 million students will participate. 


SUMMARY OF KEY FINDINGS 


With minor exceptions, the field test went very smoothly. Most students had sufficient time to complete the field test and were 
comfortable with the computer-based items, especially those students who practiced first with sample questions and tools designed 
to familiarize them with features of the online system. Likewise, schools that used the various tools, manuals, and training modules 
provided by the PARCC consortium reported fewer issues than those that did not — particularly in regard to technology. 


QUALITY OF TEST ITEMS AND STUDENT EXPERIENCE 


Item-level data from the field test indicate that a large majority of the items developed for the PARCC assessments over the past several 
years performed well. That is, students understood the questions and responded appropriately. Approximately 89 percent of the English 
language arts/literacy (ELA/L) questions and 78 percent of the mathematics questions were found eligible for the 2014—15 administration, 


mirroring results for other new assessment programs and providing ample test items for the administration. 


Generally speaking, students across grades, even those in elementary school, were able to successfully use the computer-based test 
(CBT) delivery system, including keyboarding their answers to short and extended questions, scrolling through reading passages, 
and moving from one question to the next. In addition, students told observing researchers that they found the computer-based 
assessments engaging. 


Other findings related to the student experience were: 
™ = = More than 90 percent of students said that they had sufficient time to complete the test. 


© = Approximately 90 percent of students said that they understood the directions on the ELA/L tests (both the computer- 
based and paper-based formats) and the mathematics computer-based test. Eighty-three percent of students said they 
understood the instructions on the paper-based mathematics test. 


Two-thirds of students said that they entered mathematics symbols and numbers with ease on the CBT. 


Approximately half of students said that it was easy to use the online calculator (which, nonetheless, is being revised for 
the 2014-15 administration). 
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TECHNOLOGY PREPAREDNESS 


There were no system-wide technology issues during the field test. Most technology issues that did occur were local (e.g., firewall 
settings needed to be changed, computer settings needed to be adjusted, or students needed help logging in) — an expected result 
when school districts introduce computer testing for the first time, as was the case in most PARCC states. Most issues were also quickly 


and easily resolved. 
Other findings related to technology preparedness were: 


© = =More than two-thirds of test coordinators and test administrators conducted an infrastructure trial in their district before 
the field test. 


® = Almost two-thirds of test coordinators and administrators used data from the Technology Readiness Tool to prepare for 
the field test. 


TRAINING MATERIALS AND ADMINISTRATION PROCEDURES 


In some cases, test administrators reported that some directions, such as how to close out of online test sessions, needed clarification 
and that the test manuals could be shorter and clearer. Students and test administrators and coordinators requested some improvements 


to the equation editor, a computer-based feature that allows students to solve mathematics problems and explain their reasoning. 
Other findings related to training materials and administration procedures were: 


™ = Just over half (55 percent) of test coordinators reported that the student tutorial was useful for students to become 


familiar with PARCC items, tools, and functionalities of the computer-based delivery platform. 


® = Just under half (46 percent) of test coordinators and test administrators indicated that information in the test 


administration manuals was sufficiently comprehensive. 


® = Thirty-nine percent of test coordinators agreed that the process for setting up test sessions and registering students 
through PearsonAccess (the CBT delivery system) was straightforward and simple. Forty-five percent found PearsonAccess 
easy to use. 


SUMMARY OF KEY PLANNED IMPROVEMENTS 


The PARCC states gained considerable information and insight through the administration of the PARCC field test and the analysis of 
information from student and educator surveys. The administration was successful — from the overall effective functioning of the 
testing and administration platforms to the student experience. A number of opportunities for adjustments and improvements did 


present themselves, and the PARCC consortium has been actively working to implement those improvements. 

The steps that the PARCC states are taking to help ensure the success of the 2014-15 administration and all future administrations include: 
Revising manuals and training modules; 

Revising general directions on the tests, especially the mathematics tests, to make them clearer; 

Upgrading PearsonAccess and TestNav8 (the CBT delivery platform); 

Conducting a third-party verification and validation of TestNav8 performance; 


Revising tutorials to include a full array of tools, accessibility features, and item-computer interactions; and 


Expanding practice tests to include paper-based tests and additional components (performance-based and end-of-year 


assessments in both content areas). 
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FIELD TEST BACKGROUND 


Between late March and early June 2014, 14 states and the District of Columbia administered the PARCC field test. Nearly 16,000 schools 
participated, with 73 percent participating in the test online and 27 percent in the paper-based test (PBT). Schools were randomly 
selected to participate so that information generated by the field test would be representative of the student population in the PARCC 
states. 


O> M@ Field test locations 
FIELD TEST FEEDBACK 


To evaluate the PARCC field test experience, the PARCC states conducted formal surveys of test administrators, test coordinators, and 


students. The surveys were designed to collect feedback on the training materials and manuals, as well as the student experience with 
technology and other aspects of the field test. 


Feedback also was compiled through direct observation of the field test in approximately 40 schools, direct email communication 
received by the PARCC states from field test participants, and through social media channels (i.e., Facebook and Twitter). 
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FINDINGS 


The PARCC states have now analyzed this information and have identified lessons learned and improvements to be made. This report 


summarizes the key findings and describes the specific actions that the PARCC states will take going forward. 


This level of public analysis and disclosure reflects the ongoing commitment by the PARCC states to be transparent as they develop the 


next generation of tests. 

The areas of focus for this report are: 
™ Test items 

Student experience 

Technology preparedness 

Training materials 

Manuals 


Administration procedures 


Customer support 
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Test Items 


Before being field tested, PARCC test items were reviewed by multiple committees of 
educators from elementary, middle, and high school, as well as by faculty from colleges and 
universities representing all the PARCC states. These committees checked every item to ensure: 


® Alignment to standards: Does the item measure what it is intended to measure? 
© Content correctness: Is the item free of errors? 


® Accessibility, fairness, and sensitivity: Might the item advantage or disadvantage any particular group of students? Does it 


appropriately consider the cultural backgrounds of students in the PARCC states? 
® Developmental appropriateness: Is the language and content right for students of that age? 


Every item appearing on the spring 2014 field test was reviewed by at least 30 people, many of them educators from the PARCC states, 


before being considered eligible to be placed in front of students. 


One key purpose of the spring 2014 field test was to “test the test questions” to ensure that all items that appear in the 2014-15 
assessments do a good job of measuring content, are clear, and are fair to all students. Although every item goes through multiple 
reviews prior to being field tested, the final determination regarding the quality of PARCC items is made only after they have been 


included in a test taken by students. 


Life Cycle of a Test Item 


Start with 
Standards 


Review Build the 


Develop - Draft Test Local teachers, principals, MN Field Test Test IN Administer 


Test curriculum coordinators, state , 
Design Items content experts & higher ed the Items with selected the Test 


faculty ensure that items are: items 


Reuse Release a 
Remaining Portion of 
GR the Items 


Local educators from the PARCC states are involved at every step 
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Across all grade levels and courses, the PARCC consortium field tested more than 11,000 items — approximately 6,600 mathematics 
items and 4,600 ELA/L items, embedded in more than 400 English language arts/literacy (ELA/L) tasks and text sets. 


On average, more than 1,100 students responded to each item during the field test. These responses were scored and generated 
item-level statistics such as difficulty, percentage of students who answered correctly, and whether the item seemed to advantage 
or disadvantage any particular subgroup of students. During a week-long review meeting, 80 educators from all of the PARCC states 
received training on how to interpret the item-level statistics, then reviewed the statistics for each field-tested item to determine 


whether it should be considered for use in the 2014-15 administration. 
Most items were approved through this process. Others took two paths: 


i. . Some items needed further modifications to their content or format to be usable in the 2014-15 
administration or future administrations. These revised items once again go through the PARCC item review committees, are field 


tested, and go through data review before being included in a PARCC assessment. 


2. The PARCC educators reviewing the field test data may determine that a small number of items are not candidates for 


revision and remove them from the item development process. 


The vast majority of items were approved through this rigorous review process. Approximately 89 percent of mathematics items and 
78 percent of ELA/L text sets were approved. This cycle of field testing and data review is standard for all assessment systems, and the 


approval rates are typical for test items. 


These items and text sets will be added to the PARCC item bank, which is the pool of items eligible to be used in the PARCC assessments. 


The mathematics performance-based 


assessment (PBA) has tasks that assess whether The ELA/L PBA consists of three tasks: literary analysis, 


students can apply math in real-world situations research simulation, and narrative writing. For each task, 


and reason about math concepts. Some of the students are asked to read or view one or more texts, 


tasks ask students to explain how they solved answer several short comprehension and vocabulary 


a problem or show their work. Other tasks on questions, and write an essay that requires them to 


the PBA assess conceptual understanding, skills, draw evidence from the text(s). The PARCC assessments 


and applications using multiple-choice and use both printed and multimedia texts. 


technology-enhanced items. 
The ELA/L EOY assessment consists of two to four literary 


The mathematics end-of-year (EOY) assessment and informational texts — including social science/ 


consists only of tasks that assess a balance of historical, scientific, and technical texts at grades 6-11. 


conceptual understanding, brief applications, Each text has five to six short comprehension and 


and skills and procedures. The EOY assessment vocabulary questions associated with it. A text and its 


tasks are all multiple-choice and technology- associated questions are called a “text set.” 
enhanced items and can be scored using 


technology. 
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Student Experience 


Based on the student survey results from the spring 2014 PARCC field test and observations 
by test administrators, students found the assessments more engaging than previous tests 
and had a generally positive experience with the field test. They liked the computer-based 
platform and found it easy to use. They also did not have difficulty keyboarding, a concern 
that had been raised by some educators and parents, especially for students at lower grade 
levels. Students indicated that they were generally able to understand the directions. Most 
students finished within the time allotted. Many found the test content familiar, but more 
challenging than their schoolwork, especially in mathematics. 


The PARCC consortium provided a student tutorial and practice tests to help familiarize students with the computer-based testing tools, 
the process for using the navigation within the computer-based test (CBT) delivery platform, and the process of responding to different 


types of items on the computer. 


™ 55 percent of test coordinators reported that the student tutorial helped students become familiar with the computer 


delivery platform and tools. 


™ 77 percent of test administrators reported that students in their CBT sessions practiced with the PARCC sample items 
before the field test. 


KEY FINDINGS 


The next several pages present results from the surveys students took after completing the PARCC field test. Most survey questions 
were intended for all students — those taking the CBT or PBT. Some questions were intended only for students who took the CBT, and 


the survey results indicate where there is a distinction between the CBT and PBT. 
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THE MAJORITY OF STUDENTS FINISHED WITHIN THE ALLOTTED TIME. 


The overwhelming majority of students reported that they finished within the allotted time. One of the goals of the PARCC consortium’s 
research was determining whether the time allowed for each test was sufficient for students to show what they know and can do. Using 
the results of this research, the PARCC states made some adjustments to the scheduled times for the 2014-15 administration, reducing 
overall testing time. Details may be found at www.parcconline.org/update-session-times. 


Did you have enough time to finish the test? 


Computer-Based Test Paper-Based Test 


1% 2% 1% 


ELA/L 


M™ Finished very early 


Total Responses B@ Finished on time Total Responses 
381,978 Ak 118,852 
™ Rushed to finish 
9 9 
4% Did not finish 1% 


Mathematics 


Total Responses Total Responses 
340,971 131,525 
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THE MAJORITY OF STUDENTS UNDERSTOOD THE TEST DIRECTIONS. 


Because computer-based testing is new to many students, it was important to research whether students were able to understand 
the directions. For both the paper-based and computer-based versions, the majority of students reported that they understood the 
test directions that were read to them. For the 2014-15 administration, test scripts, which are the directions read aloud by the test 


administrators, are being updated with more age-appropriate language to ensure students’ understanding of test directions. 


Did you understand all of the directions read by the person who gave the test? 


Computer-Based Test Paper-Based Test 


Total Responses Total Responses 
381,978 ®@ Yes HM No 118,852 
Mathematics 
Total Responses Total Responses 
340,971 131,525 
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STUDENTS FOUND THE MATHEMATICS EXPECTATIONS MORE CHALLENGING 
THAN THE ELA/L TEST. 


One goal of the PARCC assessments is to mirror classroom instruction that is aligned to the Common Core State Standards. The survey 


asked students to reflect on the PARCC assessments content and how it compares to what they see in their daily instruction. Overall, 


students felt that the content of the ELA/L assessments was similar to their schoolwork, while the mathematics assessments were more 


challenging than their schoolwork. This was true for both the CBT and PBT. 


How difficult was the test? 


Computer-Based Test 


ELA/L 


®@ Easier than school work 


Total Responses 
381,978 lM Same as school work 


© Harder than school work 


Mathematics 


Total Responses 
340,971 


Paper-Based Test 


Total Responses 
118,852 


4% 


Total Responses 
131,525 
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How many questions asked you about things you have not learned in school this year? 


Students responded that overall, they had been exposed to most of the PARCC assessments content in their classroom 
instruction. They reported less familiarity with content on the mathematics assessments than on the ELA/L assessments. 


Computer-Based Test Paper-Based Test 


4% 3% 


ELA/L 


© None of them 


Total Responses Mi Few of them Total Responses 
381,978 ® Most of them 118,852 
A% All of them 4% 


Mathematics 


Total Responses Total Responses 
340,971 131,525 
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THE MAJORITY OF STUDENTS FOUND THE TOOLS EASY TO USE. 


The PARCC CBT delivery system includes several built-in tools that students may use to input their answers or to help organize their 


work before answering. Overall, students responded that these tools were easy to use and that moving between items and passages 
was not challenging. 


Was it easy to use the highlighter tool? 


Students used this tool to highlight key details in test questions or passages to help them respond to test questions. 


ELA/L Mathematics 
™@ Yes 
H No 
© Did not use 
highlighter 
Total Responses Total Responses 
381,978 340,971 


Was it easy to enter math symbols and numbers for your answers? 

For the mathematics assessments, students used an “equation editor” to input answers to some of the test questions. 
While many students responded that it was easy to input their answers, updates to this tool to improve usability are 
being made for the 2014-15 administration. 


Mathematics 


®@ Yes 


HM No 
66% © Did not use mathematics symbols or numbers 


Total Responses: 
340,971 
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Was it easy to use the calculator? 


The mathematics CBTs for grade 6 through high school include a built-in grade-appropriate calculator. Students taking 
the grades 3-5 assessments do not have access to a calculator. 


Mathematics 


© Yes 
H No 


© Did not use calculator 


Total Responses: 
340,971 


Was it easy to move back and forth between passages or stories? 

Many of the tasks on PARCC ELA/L assessments require students to read multiple passages and draw evidence from 
the texts in their responses. Students overwhelmingly responded that they found it easy to navigate between the 
passages on the computer-based delivery platform. 


ELA/L 


8% 


®@ Yes 
HM No 


Total Responses: 
381,978 
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Was it easy to type your answers? 
Students who participated in the field test overwhelmingly indicated that it was easy to type the ELA/L answers, and 
approximately two-thirds said that they were comfortable typing their answers in mathematics. 


ELA/L Mathematics 


Performance-Based Assessment Performance-Based Assessment 


®@ Yes 
® No 


Total Responses Total Responses 
215,656 179,558 


PLANNED IMPROVEMENTS 


The PARCC states are updating and expanding the resources available to help students become familiar with what they can expect 
when taking PARCC assessments. Separate tutorials for each grade band (elementary, middle, and high school) are being created to 
ensure that the tutorials are grade appropriate. Additional student tutorials will cover a wider range of tools and item types. 


The planned release dates of the student tutorials for the PBT and CBT are: 
™ ~~ High school tutorials: November 2014 (available now at parcc.pearson.com/tutorial) 
™ = Grades 3-8 tutorials: January 2014 (will be posted at the same location) 
Online practice tests for the mathematics EOY assessment and the ELA/L PBA are available online now at parcc.pearson.com. 


Online and paper-based practice tests for the mathematics PBA and the ELA/L EOY assessment are in progress. 
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Technology Preparedness 


The computer-based PARCC assessments are delivered on Pearson’s TestNav8 platform, which 
is the same system used by some state assessment programs. TestNav8 is supported on a 
wide variety of devices, including desktop and laptop computers, notebooks, and tablets. 


Test administrators and coordinators and technology coordinators were asked about their experiences with TestNav8, including logging 
students into the system and carrying out other administrative functions. 


The PARCC consortium made several tools and activities available to districts and schools to help them prepare for the field test and 
test their local technology before the field test. These included: 


© The infrastructure trial, which gave local school districts, schools, and students the opportunity to prepare for the 
computer-based PARCC field test by simulating test-day network use; 


™ = The SystemCheck tool, which allowed users to validate that their testing workstations met the minimum requirements 
needed to run TestNav8 for the field test and to evaluate bandwidth capacity for internet and proctor-caching connections; 


™ = The Technology Readiness Tool (TRT), which evaluated and determined necessary technology and infrastructure upgrades 


for the new online assessments; and 


™ Proctor caching, which pulled and stored test content from Pearson to a local computer and distributed this "cached" 
content to students in order to increase speed of delivery and reduce bandwidth usage. (See next page.) 


KEY FINDINGS 


Based on survey feedback, the majority of test administrators and test coordinators took advantage of the technology preparedness 
tools and activities. 


™ 69 percent of test coordinators and test administrators conducted an infrastructure trial. 


™ 62 percent of test coordinators and test administrators used data from the TRT to evaluate whether the number of devices 
and bandwidth were sufficient to administer the test. 


™ 60 percent of test coordinators and test administrators used proctor caching. 


Even with the high use of the available technology preparedness tools, some schools indicated that they experienced local technology 
issues, for example devices that stopped working, devices that worked slowly, or cases where internet connection was lost during 


administration. 


16 Ml PARCC Field Test: Lessons Learned 


PLANNED IMPROVEMENTS 


Although local school districts are responsible for their own technology (devices, internet connectivity, etc.), the PARCC consortium has 


worked since the field test to provide districts and schools with additional supports to improve the local technology experience. 


™ = The PARCC states made another infrastructure trial available to schools in October, providing them with ample time to 
conduct this test-day network use simulation prior to the 2014-15 administration. Schools and districts that had conducted 


an infrastructure trial had more seamless experiences during the field test. 


™ The PARCC states are enhancing the guidance and documentation around the SystemCheck tool to ensure wider use and 
understanding. SystemCheck allows customers to validate that their testing workstations meet the minimum requirements 


needed to run TestNav8 for the assessments and evaluate bandwidth capacity for internet and proctor-caching connections. 


™ The PARCC states are updating the training modules available to guide users in conducting an infrastructure trial and 


proctor caching to ensure that they are more comprehensive and provide clearer step-by-step instructions. 


™ The PARCC states will provide a new technology preparedness training module to help organizations understand, manage, 


and make decisions in preparing school technology to be used for online testing. 


™ The PARCC states are developing clear, consistent, and comprehensive communication to be sent to states and districts 


regarding technology preparedness tools, resources, timelines, and support. 


In order to address bandwidth challenges in some school districts, the PARCC states provided 

a low-bandwidth option for computer-based testing that loads tests faster for students and 
minimizes the impact of computer-based testing on district bandwidth use. Proctor caching is 
a method by which test content is stored to a local computer and then distributed to students 
taking the test. Districts had the flexibility to determine where to implement proctor caching in 
their network environment. Based on local network considerations, districts could implement 
proctor-caching machines at the district, school, or classroom level. By implementing proctor- 
caching software, districts and schools were able to download one copy of the test to a specific 
computer; then, during testing, content was sent directly from this caching computer to the 
students’ test-taking computers. 


Approximately 60 percent of schools and districts took advantage of proctor caching, according 
to the technology coordinator survey results. In general, technology coordinators found proctor- 
caching guides and tutorials to be helpful in setup and implementation. Overall, they found the 
hardware requirements easy to meet and were pleased that the software could run on many 
desktop computers. One IT manager in Massachusetts even wrote a blog post documenting the 
ease of his experience with the proctor-caching setup (bit.ly/PARCCguestblog). 


Updated proctor-caching software is now available at parcc-test.pearson.com/technology-setup, with an updated interface to make it more user 


friendly. This update will not be required for schools that already set up proctor caching for the field test, but it is strongly recommended, as it will 


improve performance. Enhancements include a simplified user interface, resolution for Internet Explorer 11 connectivity issues, and improved 
reporting to help verify that the most up-to-date content is available for testing. 


The PARCC states are also reviewing the proctor-caching documentation and training materials to help users understand how to monitor and 
manage proctor caching. In addition, the PARCC states will continue to encourage the use of infrastructure trials to help schools and districts 
better understand the proctor-caching process and prepare for test day. 
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Training Materials 


Training support was provided to test and technology coordinators and other staff in states, 
local districts, and schools in two formats: online training modules and student tutorials. Test 
coordinators and administrators said that the training was helpful in creating a smoother 
experience during the field test. 


In preparation for field testing, the PARCC consortium worked with Pearson to provide a variety of on- and off-site trainings for PBA and 
EOY assessment participants. These trainings included: 


® 36 regional training workshops; 
©  S4site visits; 
™ 7 question-and-answer workshops; 
™ 1student tutorial and several practice tests; and 
® = 10 online training modules. 
The following online training modules were offered in advance of the PARCC field test: 


SystemCheck tool. This module contains information about using the SystemCheck tool to review device and network readiness. It 
provides information about the content and usage of the two available tools, as well as examples of error messages and of successful 
completion of a device check and a network capacity check. 


Student Data Upload Overview.* This module provides step-by-step instructions for states, districts, and schools to load students into 
PearsonAccess using the PARCC Student Data Upload file layout. 


Setting Up an Infrastructure Trial/Dress Rehearsal. This module provides a description of an infrastructure trial and information on 
how to prepare for one using the PARCC Training Center. 


Technical Setup. This module describes each of the technology components used in computer-based testing. Configuration and 
hardware requirements are included, as are common best practices for successfully testing online. Users are provided with information 
on accessing separate training modules for SystemCheck, emerging technologies, and setting up an infrastructure trial to avoid 


redundancy in training. 


Test Administration for Computer-Based Testing. This module covers the tasks and activities for test administration staff using 
PearsonAccess. Information is provided on managing user accounts and student data, as well as test management. Viewers also receive 


information for creating and managing online sessions during field test administration. 


Test Administration for Paper-Based Testing. This module covers the tasks and activities for test administration staff using 
PearsonAccess. Information is provided on managing user accounts and student data, as well as test management. 


Emerging Technologies and Security with Computer-Based Testing. This module addresses special considerations with iPads and 
virtualization solutions. New technologies present additional security challenges, which are addressed along with other security 
concerns specific to computer-based testing. Those include active test monitoring, securing authorization tickets, and computer lab 
physical configuration. 


1 PearsonAccess Next refers to the Student Data Upload (SDU) as Student Registration Import, which is the term that will be used going 
forward. 
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Accessibility Features and Accommodations with Computer-Based Testing. This module is designed to provide information about 
available accessibility features and accommodations for computer-based testing. It references the Draft Accommodations manual and 
Full Specifications document available on the PARCC website. This module also covers the steps to assign an accommodated form to a 


student in the Create Sessions process within PearsonAccess. 


Training for Test Coordinators. This module provides an overview of information covered in the test coordinator manuals, such as roles 


and responsibilities of test coordinators and tasks to complete before, during, and after administration of the PARCC assessments. 


Training for Test Administrators. This module provides an overview of information covered in the test administrator manuals, such as the 


roles and responsibilities of test administrators and tasks to complete before, during, and after administration of the PARCC assessments. 


KEY FINDINGS 


The majority of test administrators and test coordinators (on average, 59 percent) indicated that the online training modules were 


useful in preparing and administering the assessments. 
According to test administrators, the most viewed PARCC training modules were: 
® Test Administration for Computer-Based Testing 
® = Accessibility Features and Accommodations with Computer-Based Testing 
™ = Setting Up an Infrastructure Trial/Dress Rehearsal 
According to test coordinators, the most viewed PARCC training modules were: 
© Training for Test Coordinators 
™ Setting Up an Infrastructure Trial/Dress Rehearsal 


© Training for Test Administrators 


IMPROVEMENTS 


The PARCC states have made several improvements to available modules based on the field test feedback, including: 
™ Student Registration Import (September 19, 2014) 

PearsonAccess Next (October 9, 2014) 

Paper-Based Testing for Test Coordinators (October 10, 2014) 


Paper-Based Testing for Test Administrations (October 13, 2014) 


Introductions to Training (October 20, 2014) 
© Accessibility Features and Accommodations (October 21, 2014) 


Modules now make clearer which tasks pertain to test administrators and which to test coordinators. 
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In preparation for field testing, the PARCC states provided test administration manuals. 


The manuals used during the field test were: 
™ Computer-Based Testing Field Test: Test Administrator Manual 
™  Paper-Based Testing Field Test: Test Administrator Manual 
™ Computer-Based Testing Field Test: Test Coordinator Manual 
| 


Paper-Based Testing Field Test: Test Coordinator Manual 


KEY FINDINGS 


Survey questions helped capture school and district test coordinators’ overall opinions on the quality of the test administration 


manuals, including whether they were user friendly and contained sufficient information for test administration. 


SURVEY STATEMENT Strongly Agree/ Strongly Disagree/ 
Agree Disagree 
The test administration manuals were user friendly/ 
; 42% 19% 29% 
information was easy to find. 
Language [in the test administration manuals] was 
42% 20% 27% 
clear and concise. 
Information [in the test administration manuals] 
50% 23% 18% 
was relevant and useful. 
Information [in the test administration manuals] 
es 46% 21% 23% 
was sufficiently comprehensive. 


*Note that percentages do not add up to 100 because the remaining respondents selected “Not Applicable.” 
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PLANNED IMPROVEMENTS 


Many of those who responded to the test administrator and test coordinator surveys called for improvements to the test administration 


manuals and procedures. 


SIMPLIFYING MANUALS AND MAKING THEM GRADE APPROPRIATE 


The PARCC states: 
™ Have streamlined and simplified the manuals, including administration scripts; 


® = Are working with grade-level experts to ensure that administration scripts are grade appropriate, as the manuals will be 


split into grade bands; 


Are updating manuals to include a simple and direct checklist of tasks to complete before, during, and after testing; and 


Are updating manuals to include more explicit directions, including graphics and icons to increase user understanding. 


CLARIFYING MAKE-UP TESTING POLICIES 
The PARCC states: 


® = = Are working to simplify and clarify make-up testing policies, including allowing more flexibility for scheduling purposes; and 


® Are carefully reviewing and updating manuals to ensure that procedures and directions are simple and clear. 


IMPROVING CALCULATORS 


™ = The PARCC states are working on enhancements in the calculator sections to make it easier for test administrators to 
monitor calculator use, including: 
> For computer-based testing, changes in the interface will make it easier to monitor whether a student is in a calculator 
section. 


> For paper-based testing, a calculator icon will be added to the pages in the calculator sections of the test booklet. 


ADJUSTING SUBMIT PROCESS 


™ The PARCC states will only make the “Submit” function available in the final section of the exam. Prior to that, only “Exit & 


Save” can be selected to leave the test. 
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To administer the field test, participants used the following platforms. 
™  PearsonAccess, the single, consolidated online system for managing the PARCC assessments. 


>  PearsonAccess is role-based, meaning users will see only those data and perform only those tasks for which they have 
been authorized. 


> Based on a user’s role in the system, he or she may do one or more of the following: manage student data for testing, 
manage staff accounts, and/or create CBT sessions. 


™  TestNav8, a secure, browser-based application used at student workstations to take CBTs. 


>  TestNav8 runs in the cloud, delivering tests via a web browser on desktop and laptop computers and via a custom app 
on selected tablets. 


KEY FINDINGS 


Survey questions helped capture school and district test coordinators’ overall opinions on the quality of the test administration 


procedures, including whether they were user friendly and contained sufficient information for test administration. 


SURVEY STATEMENT gly Disagree/ 
Disagree 

The PearsonAccess portal used by PARCC was easy 

45% 19% 20% 
to use. 
The student registration process was 

: 37% 17% 16% 

straightforward and easy to complete. 
The task required for the test setup process was 

39% 19% 17% 
straightforward and easy to complete. 
TestNav8 worked well during test administration. 28% 15% 40% 


*Note that percentages do not add up to 100 because the remaining respondents selected “Not Applicable.” 
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PLANNED IMPROVEMENTS 


The PARCC consortium’s plan for responding to the feedback is outlined below. 


USER FRIENDLINESS 


® A revised version of PearsonAccess called PearsonAccess Next was launched this fall with an updated interface to make it 


more user friendly. 


® = To avoid confusion, the training site and the site where information is formally entered will have different color schemes. 


The training site will also be clearly labeled with a large header. 


™ With the launch of PearsonAccess Next, system processes will be updated to be task-based, ensuring that they are more 


streamlined and less time consuming. 


™ By design, PearsonAccess Next times out after 15 minutes for security purposes. The consortium will communicate this 


requirement to users. 


STUDENT REGISTRATION FILE AND IMPORT 


™ To allow users to make changes to existing student records, the PARCC consortium provided the Student Registration File 
layout, including field definitions, at the end of August 2014. 


™ The Student Registration window was open from early September through early October for the fall block administration. 


® For the 2014-15 administration, users will have the option to auto-create test sessions in PearsonAccess Next for all 


students in a classroom at once based on class name. 


TEST MANAGEMENT IMPROVEMENTS will clarify the process for setting up test sessions, make it easier to 


manage absent students, and move students to different test sessions within PearsonAccess Next. 


™ = PearsonAccess Next will allow for the automatic creation of test sessions based on the information in the Student 


Registration File. 


™ For spring 2015 and beyond, only the number of seal codes needed for a given test (per grade level) will be made available 


to test administrators. 


™ = Manuals and guides were updated to include clear instructions on how to create and manage sessions along with 


managing students (including absent students) beginning with the fall block. 


™  PearsonAccess Next configuration will allow a student to be moved from read-aloud to non-read-aloud without having to 


set up a new test session. 


ONLINE TESTING IMPROVEMENTS will adaress performance issues with some tools, problems loading the 


videos, and some students being locked out of sessions. 


The PARCC states will use an independent vendor to test the platform and identify performance issues needing improvement. 
The PARCC states will also conduct “load testing” and other tests to pinpoint performance issues and solutions. 


® As part of the January release of TestNav8, the equation editor will be updated with a simplified user interface, improved 


usability of mathematics functions, and improved tablet usability. 
As of the October release of TestNav8, the “Submit” button is only available in the final section of the exam. 


The PARCC states will improve information dissemination on minimum technology specifications and how to set up devices. 
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Leading up to and during the field test administration, a call center provided technical 
assistance in many areas, including: 


™ Preparing for proctor caching; 
Registering students; 


| 

™ = Student login troubleshooting; 

™ Firewall settings and other local technology issues that interfered with signing students on; and 
| 


Resets when students inadvertently submitted their answers before completing a test section. 


KEY FINDINGS 


The volume of calls to the support center in the early days of the field test exceeded expected levels and resulted in slow response 
times and dropped calls. More than 1,300 people called or emailed the call center each of the first two days of the field test. It took 
more than a minute the first day and close to a minute the second day for calls to be answered. More than 80 people abandoned their 
calls over the first two days. 


The PARCC consortium worked with Pearson to quickly and significantly increase the number of support specialists and to improve their 
training. Within days, with more staff at the call center and better training, calls were answered, on average, within three seconds, and 
on most days no callers abandoned their calls. 


When asked about their experience with the Pearson-run PARCC support center during the field test, test coordinator survey 
respondents indicated that their service requests were answered promptly 75 percent of the time, their questions were answered 
within one communication 65 percent of the time, and their questions were answered accurately 51 percent of the time. 


PLANNED IMPROVEMENTS 


Support for field test participants was an important success factor and will continue to play an integral role in the PARCC fall block and 
spring administrations. Below is a summary of how the PARCC consortium plans to respond to the feedback regarding customer service. 


® = Call center agents will be trained on basic customer service; testing tools such as TestNav8, PearsonAccess Next, 
SystemCheck, and training management site; and tablet training. 


™ ~The PARCC consortium created a Quick Start Checklist based on feedback from test coordinators indicating the need for a 
cohesive step-by-step guide. This gives test coordinators and technology coordinators an overview of the entire process, 
with references and links to the various tools and resources available to them. 
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Social media played an important role during the field test. The PARCC consortium had more than 14,000 Twitter 


followers during the field test (and more than 17,000 at the issuance of this report). Social media allowed the 
PARCC consortium to see how some teachers share resources, how tech coordinators share helpful tips, and 


how test administrators seek advice from each other. During field testing, PARCC: 
Posted live updates to the PARCC online news page; 
Tweeted important information each day; 


Asked participants to share their field test experience using the hashtags #PARCCfieldtest and #askPARCC; 


and 


Asked test coordinators and administrators to send direct messages to the PARCC consortium via Twitter 


with questions. 


Unlike surveys, social media provided the PARCC consortium with lessons learned in real time. The PARCC 
consortium was able to view participant feedback instantly through live posts and develop a plan of action to 
address that feedback. Social media will be monitored in future administrations of the assessments as well. 


Sample Tweets from the Field 


@Susan_Kahn thank you! Can honestly say my son's experience with #PARCC 
#PARCC field resting went well 


today. Day 2 for 3rd graders. No tech 


reaffirmed my belief in this assessment hundred fold #coreadvocates 


issues. Tech staff was awesome! Grt 
team. #edtechchat #satchat #njed 


#PARCCfieldtest #askPARCC 


#parcc #parccfieldtest Seems that turning off 


anti-virus eliminates all of the problems with 


Watching the demo of What a pleasure to spend 


PHATAPARGGtesthnave video load we were experiencing. @PARCCPlace aaae wlth tae SO Gln 

video questions!! Very graders who took the 

cool #edchat PARCC Field Test the past 
PARCC Field Test: PBA Field Test complete. With few days. They enhanced 


the exception of minor blip day1, all went very my faith in the future. 


well. Planning & collaboration = success! 


Pearson and the PARCC consortium were able to take steps to improve customer service during the field test 
by monitoring social media and the live feedback received from field test participants. They provided real-time 


assistance by reaching out directly to schools that needed it. 
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Post-Field Test Timeline 


The timeline below details the major activities associated with the 2014-15 PARCC 


assessments. 
SPRING FALL 
2015 2015 


Analysis and 
research Lessons 


learned 
rele] ) [acre] 


Data review PARCC research reports available 
Test construction and forms review 
Fall testing 
window* 
Updated fall Spring testing 
block manuals window 


Updated fall block online training modules 


Scoring and 
analysis 


Standard 
setting 


Updated student tutorials 


Updated practice tests Reporting 


Updated 
spring 
manuals 


*High schools with block scheduling 


Note: These are estimated timeframes and subject to change. 
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As planned, the field test provided an opportunity to test the test items, evaluate the 
administration process, and conduct research. Because the field test administration was 
successful overall, the PARCC consortium has been able to focus on making important 
refinements and improvements. 


Evidence gathered during the field test and from survey responses indicated that the test content hit the mark. Most students were 
able to follow instructions, use the keyboard and various test features (such as the online calculator), and complete the tests in less 
than the time allotted. 


The online test system performed well. Where technology issues appeared, they were at the local level. The PARCC states are adding 
resources to help schools and districts, especially those that are not experienced in computer-based testing, prepare for the 2014-15 
administration. 


The PARCC states are making improvements to test administration manuals and some administrative features and processes. The 
states are making it easier for students to correctly exit test sessions and use the mathematics equation editor and clarifying when 


the online calculator can be used. 


In addition, the PARCC states are preparing to release practice PBTs and additional online practice tests to provide more resources for 
teachers and students. 


The field test experience showcased the power of states working together. It was through the contributions and expertise of teachers, 


principals, curriculum coordinators, superintendents, and others from multiple states that this undertaking was possible. 


If you have any questions, please contact PARCC at questions@parcconline.org. 


To stay connected, sign up for PARCC updates at parcconline.org and follow PARCC on Facebook and Twitter. 
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PARCC Terms and Acronyms 


TERMS 


Field test: A test that evaluates the PARCC assessments by helping to ensure that they are valid, reliable, and fair. 


Infrastructure trial: A trial that provides the opportunity for districts, schools, and students to prepare for the computer-based PARCC 
field test by simulating test-day network use. 


PearsonAccess: The administrative website used by the district and by school coordinators, administrators, and technology 


coordinators. 
PearsonAccess Next: The test-management system that replaced PearsonAccess in fall 2014. 


Proctor caching: A process that pulls and stores test content from Pearson to a local computer. This “cached” content is distributed to 
TestNav8 clients in order to increase the speed of delivery and reduce bandwidth usage. 


Seal codes: The electronic equivalents of the adhesive tabs that are used to seal sections of paper test booklets. There will be one set 
of seal codes assigned to each test session. 


Section: A portion of a mathematics unit — calculator section and non-calculator section. 


Session: A session includes all of the units for a subject and may be scheduled across one or more days. Sessions are scheduled by 
subject and the group of students testing that subject together (as set up in PearsonAccess Next for computer-based testing). 


SystemCheck tool: A tool that allows customers to validate that their testing workstations meet the minimum requirements needed to 
run TestNav8 for the field test and evaluate bandwidth capacity for internet and proctor-caching connections. 


Technology Readiness Tool (TRT): A tool that evaluates and determines needed technology and infrastructure upgrades for the new 


online assessments. 
TestNav8: The secure, browser-based student application for accessing the computer-based assessments. 


Tutorial and practice tests: Tools that provide the opportunity for districts, schools, and students to familiarize themselves with the 
design and functionality of online testing. 


ACRONYMS 


CBT — computer-based test 

ELA/L - English language arts/ literacy 

EOY - end of year 

PARCC — Partnership for the Assessment of Readiness for College and Careers 
PBA — performance-based assessment 

PBT — paper-based test 


TRT — Technology Readiness Tool 
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